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LONG-TERM  GOAL 

The  long-term  goal  of  this  research  is  to  construct  the  Navy’s  next  generation  global  numerical 
weather  prediction  (NWP)  model  using  new  numerical  methods  specifically  for  distributed-memory 
computers.  To  take  full  advantage  of  the  new  computer  architectures,  the  spherical  global  domain  must 
be  partitioned  into  local  sub-domains,  or  elements,  which  can  then  be  solved  independently  on  the 
multiple  processors  of  these  computers.  The  numerical  methods  used  on  these  sub-domains  must  be 
not  only  local  in  nature  but  also  high-order  in  accuracy  and  highly  efficient.  Thus  the  final  objective  of 
this  project  is  to  construct  a  new  global  NWP  model  which  is  as  accurate  as  the  current  spectral  model 
(NOGAPS)  while  much  more  efficient,  thereby  allowing  for  finer  resolution  forecasts. 

OBJECTIVES 

The  objective  of  this  project  is  to  construct  high-order  local  methods  for  the  Navy’s  next  generation 
global  NWP  model.  The  high-order  accuracy  of  these  methods  will  ensure  that  the  new  model  yields 
the  same  accuracy  as  the  current  spectral  model  while  the  locality  of  these  methods  will  ensure  that  the 
efficiency  of  the  model  increases. 

APPROACH 

To  meet  our  objectives  we  explore: 

1 .  spectral  element  methods  (SEM), 

2.  spectral  elements  in  space  with  a  semi-implicit  semi-Lagrangian  scheme  in  time  (SESL), 

3.  Fast  matrix  solvers  on  distributed-memory  computers. 

The  power  of  spectral  element  methods  is  that  they  are  high  order  accurate,  like  spectral  transform 
methods  (i.e.,  spherical  harmonics),  yet  are  completely  local  in  nature  -  meaning  that  the  equations  are 
solved  independently  within  each  individual  element  and  processor.  Semi-Lagrangian  methods  are  also 
being  considered  because  these  methods,  like  high-order  methods,  have  minimal  dispersion  errors. 

This  property  is  important  for  properly  capturing  atmospheric  phenomena.  In  addition,  semi- 
Lagrangian  methods  offer  vast  improvements  in  efficiency  due  to  the  longer  time  steps  that  they 
permit. 
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After  validation  of  the  spectral  element  and  semi-Lagrangian  discretization  methods,  the  vertical 
integration  scheme  for  the  full  3D  NWP  model  was  included.  The  hydrostatic  3D  SEM  primitive 
equation  model  was  then  ported  to  the  message-passing  interface  (MPI)  to  show  exactly  what  types  of 
performance  we  can  expect  with  these  new  high-order  local  methods.  The  final  stage  for  the  dynamics 
is  to  include  performance-enhancing  features  such  as  Helmholtz  solvers  for  the  semi-implicit  semi- 
Lagrangian  method  and  port  the  SESL  scheme  to  MPI. 

WORK  COMPLETED 

To  test  the  discretization  methods  (SEM  and  SESL)  we  have  applied  them  to  the  shallow  water 
equations  on  the  sphere.  These  methods  have  been  tested  extensively  and  three  papers  have  been 
written  concerning  these  innovative  approaches  (Ref.  [1,3,4]).  The  new  spectral  element  3D  NWP 
model  is  undergoing  extensive  validation  and  performance  testing.  Six  test  cases  have  been  applied  to 
the  new  3D  model  and  this  work  was  presented  at  a  conference  in  Toronto  (Ref.  [5])  and  at  invited 
talks  at  NCAR  (August  2002),  NCEP  and  NRL-DC  (September  2002). 

RESULTS 

Shallow  Water  Equations.  To  show  the  accuracy  of  the  spectral  element  discretization  method  we 
show  results  for  nonlinear  zonal  geostrophic  flow  (Case  2  in  Ref.  [1]).  Figure  1  shows  the  spectral 
convergence  of  the  method. 
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Figure  1:  L2  error  as  a  function  of  basis  function  order  N  for  the  spectral  element  method  (red 
line),  Japanese  earth  simulator  finite  volume  code  (red  dot),  CSU  finite  volume  code  (purple  dot), 

and  the  NCAR  spectral  transform  model  (green  dot). 


Note  the  high-order  spectral  accuracy  achieved  by  the  SEM  model.  In  addition,  this  figure  shows  that 
the  SEM  model  is  currently  the  most  accurate  numerical  scheme  as  is  evident  by  comparing  this  model 
to  three  other  models:  2  finite  volume  models  (red  and  purple  dots)  and  a  spectral  transform  model 
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(green  dot).  This  pattern  is  repeated  for  many  other  test  cases  and  in  Ref.  [5]  we  show  that  the  spectral 
element  method  is  indeed  more  accurate  than  all  currently  existing  methods. 

A  similar  result  is  obtained  with  SESL  except  that  now  we  can  use  time-steps  10  times  larger  with 
SESL  as  compared  to  SEM. 

3D  NWP  Equations.  As  a  first  test  to  validate  the  new  3D  NWP  model  Rossby-Haurwitz  waves 
number  1  and  4  were  run  both  with  the  spectral  element  Eulerian  atmospheric  model  (SEE- AM)  and 
NOGAPS.  These  test  cases  were  run  for  5  days  using  a  T80  horizontal  resolution  with  24  vertical 
levels. 
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Figure  2:  Surface  contours  for  Rossby-Haurwitz  wave  4  for  SEE-AM 
with  T80  L24  for  a  5  day  integration. 
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Figure  3:  Surface  contours  for  Rossby-Haurwitz  wave  4  for  NOGAPS 
with  T80  L24  for  a  5  day  integration. 


Note  that  the  results  between  SEE-AM  and  NOGAPS  are  identical.  This  is  also  true  for  the  wave  1 
result  (not  shown).  In  addition,  the  spectral  element  semi-Lagrangian  atmospheric  model  (SESLAM) 
also  gives  similar  results  for  these  test  cases. 

As  a  second  test,  the  SEE-AM  model  was  run  with  the  Held-Suarez  test  case.  This  case  is  the  primary 
test  run  by  all  new  atmospheric  models  because  it  mimics  a  realistic  atmosphere  with  full  physics. 
Figure  4  shows  the  contour  plot  of  the  mean  zonally-averaged  zonal  velocity  as  a  function  of  latitude 
and  vertical  coordinate  for  T64  L20  for  a  1200  day  integration. 


4 


Figure  4:  Zonally-averaged  zonal  velocity  (U)  for  the  Held-Suarez  test  case  with  T64  L20  as  a 
function  of  latitude  (theta)  and  vertical  coordinate  (sigma)  for  a  1200  day  integration. 


Figure  4  compares  well  with  the  results  of  Held-Suarez  and  those  from  the  ECMWF  spectral  transform 
model  and  clearly  shows  the  formation  of  the  jet  stream  (in  the  mid-latitudes).  This  result  shows  that 
the  model  is  stable  for  very  long-time  integrations  while  conserving  mass  and  energy. 

As  a  third  test  a  baroclinic  instability  test  was  run.  This  case  is  a  two  part  test:  in  the  first  part,  the 
model  is  run  for  30  days  to  ensure  that  the  model  retains  the  initial  balanced  state.  The  second  part  of 
the  test  involves  adding  a  perturbation  to  this  initial  state  and  then  tracing  the  evolution  of  the 
baroclinic  instability.  SEE-AM  compared  well  with  the  NCAR  spectral  transform  model  (not  shown). 

Finally,  to  show  the  scalability  of  the  new  model  we  show  a  plot  comparing  NOGAPS  to  SEE-AM  for 
a  1  day  integration  using  the  same  time-step  with  the  operational  resolution  T160  L24. 
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Figure  5:  Scalability  of  the  SEE-AM  model  (solid  blue  line)  compared  to  NOGAPS  (dotted  red  line). 
The  SEE-AM  model  can  use  many  more  processors  than  NOGAPS. 


This  result  clearly  shows  that  SEE-AM  can  not  only  use  far  more  processors  than  NOGAPS  but  that 
SEE-AM  also  scales  better  than  NOGAPS  on  a  per  processor  basis.  At  this  spatial  resolution  SEE-AM 
can  easily  use  10,000  processors  while  NOGAPS  can  only  accommodate  120  processors  efficiently. 
Very  few  models  will  be  able  to  scale  at  the  rate  that  SEE-AM  is  scaling. 

IMPACT 

NOGAPS  is  run  operationally  by  FNMOC  and  is  the  heart  of  the  Navy’s  operational  weather 
prediction  support  to  nearly  all  DOD  users  worldwide.  Our  work  here  targets  the  next  generation  of 
this  system  for  the  next  generation  of  computer  architectures.  These  architectures  are  expected  to  be 
distributed-memory,  commodity  based  systems.  The  new  model  has  been  designed  specifically  for 
these  types  of  computer  architectures  while  yielding  the  same  high-order  accuracy  as  the  current 
model. 

TRANSITIONS 

Improved  algorithms  for  model  processes  will  be  transitioned  to  6.4  (PE  0603207N)  as  they  are  ready, 
and  will  ultimately  be  transitioned  to  FNMOC  with  future  NOGAPS  upgrades. 

RELATED  PROJECTS 

Some  of  the  technology  developed  for  this  project  will  be  used  immediately  to  improve  the  current 
spectral  formulation  of  NOGAPS  (NRL  BE-35-2-18). 

In  addition,  the  work  herein  will  complement  the  work  done  on  constructing  an  efficient  message 
passing  NOGAPS  as  is  being  done  in  6.3  Common  High  Performance  Computing  Software  Support 
Initiative  (CHSSI)  (PE  0603704N). 
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SUMMARY 


We  have  constructed  a  spectral  element  Eulerian  atmospheric  model  (SEE-AM)  and  a  spectral  element 
semi-Lagrangian  atmospheric  model  (SESLAM).  Both  models  have  been  shown  to  yield  spectral 
(exponential)  convergence  and  are  currently  the  most  accurate  methods  for  solving  atmospheric  flows. 
This  high  accuracy  has  been  shown  using  shallow  water  and  full  3D  atmospheric  test  cases  which 
show  that  the  models  compare  well  with  NOGAPS  and  the  NCAR  spectral  transfonn  model.  In 
addition,  the  SEE-AM  model  has  been  ported  to  distributed-memory  computers  and  has  been  shown  to 
scale  linearly  on  these  computers  and  will  be  able  to  outpace  any  model  on  these  computer 
architectures  for  the  foreseeable  future.  The  goal  for  next  year  is  to  include  physical  parameterization 
into  the  model  in  order  to  run  more  realistic  weather  problems. 
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